Animated Lombard speech: Motion capture, facial animation and visual intelligibility of speech produced in adverse conditions

In this paper we study the production and perception of speech in diverse conditions for the purposes of accurate, flexible and ighly intelligible talking face animation. We recorded audio, video and facial motion capture data of a talker uttering a set of 180 hort sentences, under three conditions: normal speech (in quiet), Lombard speech (in noise), and whispering. We then produced an nimated 3D avatar with similar shape and appearance as the original talker and used an error minimization procedure to drive the nimated version of the talker in a way that matched the original performance as closely as possible. In a perceptual intelligibility tudy with degraded audio we then compared the animated talker against the real talker and the audio alone, in terms of audio-visual ord recognition rate across the three different production conditions. We found that the visual intelligibility of the animated talker as on par with the real talker for the Lombard and whisper conditions. In addition we created two incongruent conditions where ormal speech audio was paired with animated Lombard speech or whispering. When compared to the congruent normal speech ondition, Lombard animation yields a significant increase in intelligibility, despite the AV-incongruence. In a separate evaluation, e gathered subjective opinions on the different animations, and found that some degree of incongruence was generally accepted. 2013 Elsevier Ltd. All rights reserved.